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The Nation's air traffic controllers play an important role in maintaining the 
world's safest air transportation system. Yet, losses of standard separation — when 
aircraft do not maintain the minimum distance apart — remain a significant safety 
concern. In January 2011, an operational error — a loss of standard separation 
caused by air traffic controllers 1 — led to a near mid-air collision between a 
commercial airliner and two military aircraft near New York City. According to 
the National Transportation Safety Board (NTSB), who investigated the incident, 
at their closest point, the aircraft came within a mile of each other. According to 
Federal Aviation Administration (FAA) statistics, the number of reported 
operational errors increased by more than 50 percent between fiscal years 2009 



Concerned with this increase, the Senate Committee on Commerce, Science, and 
Transportation and the Subcommittee on Aviation Operations, Safety, and 
Security requested that we review FAA's ongoing efforts to assess operational 
errors and mitigate their risks. We also received a similar request from the House 
Committee on Transportation and Infrastructure. Accordingly, we 
(1) identified the reasons for the increase in losses of separation — specifically 
operational errors — from fiscal years 2009 to 2010; (2) assessed the effectiveness 
of FAA's policies and processes to collect, investigate, and report separation 
losses; and (3) evaluated the effectiveness of FAA's policies and processes to 
mitigate the risk of separation losses. 



1 Most losses of separation are classified as either an operational error (if the controller's actions caused the loss) or a 
pilot deviation (if the pilot's actions caused the loss). 
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We conducted this audit in accordance with generally accepted Government 
auditing standards. Exhibit A details our scope and methodology. Exhibit B lists 
the organizations we visited or contacted. 

RESULTS IN BRIEF 

According to FAA, the dramatic increase in reported operational errors between 
fiscal years 2009 and 2010 was mostly due to increased reporting through 
programs such as the Air Traffic Safety Action Program (ATSAP) and the Traffic 
Analysis and Review Program (TARP), an automated system to detect losses of 
separation at air traffic terminal facilities. 3 However, we found that the increase in 
reported errors was linked, in part, to a rise in actual errors rather than increased 
reporting. For example, FAA's air route traffic control centers (ARTCC) 4 — which 
have had an automated system in place for years to detect and investigate reported 
errors — had a 39 percent increase in operational errors during the same period. In 
addition, we identified other contributing factors to the rise in the number of 
operational errors. For example, almost one-quarter of the increase is due to the 
revocation of a separation waiver at the Southern California Terminal Radar 
Approach Control (TRACON) that led to the reclassification of many routine 
approach and landings as operational errors. 5 

FAA's new policies and procedures for collecting, investigating, and reporting 
separation losses have the potential to reduce losses and improve reporting, but 
their effectiveness is limited by incomplete data and implementation challenges. 
Under FAA's new policies, FAA uses TARP to detect losses, then examines the 
risk of these losses, and identifies corrective actions. FAA's Air Traffic 
Organization (ATO) has three regional air traffic Service Areas that review TARP 
alerts to determine whether a valid loss of separation occurred. However, FAA 
currently reviews only a portion of all TARP alerts. In addition, ATSAP data 
indicate that more operational errors might be occurring than those that FAA 
reports. As we reported last year, 6 approximately 50 percent of all ATSAP event 
reports 7 are classified as "unknown," which means that the event captured in the 



2 ATSAP is a voluntary, non-punitive program in which controllers can self-report safety instances and concerns. In 
July 2012, we issued a separate report on FAA's implementation of ATSAP (OIG Report Number 2012-152, "Long- 
Term Success of ATSAP Will Require Improvements in Oversight, Accountability, and Transparency" July 19, 2012). 
OIG reports are available on our Web site at http://www.oig.dot.gov/ . 

3 Terminal facilities include airport traffic control towers and Terminal Radar Approach Control (TRACON) facilities. 
Air traffic control towers separate aircraft on the airport surface and guide aircraft as they take off and land. TRACONs 
guide aircraft as they approach or leave airspace surrounding airports to about 40 miles away. 

4 ARTCC guide aircraft flying at high altitudes, generally above 17,000 feet. 

5 The waiver allowed aircraft landing simultaneously to be closer than normally allowed. Air Traffic Safety Oversight 
Service revoked the waiver because it considered it unsafe, and subsequently, reclassified aircraft landings that 
occurred under the waiver as operational errors. 

6 OIG Report Number 2012-152, "Long-Term Success of ATSAP Will Require Improvements in Oversight, 
Accountability, and Transparency," July 19, 2012. 

7 Event reports identify actual or potential losses of separation, including operational errors, or other situations that may 
degrade air traffic safety. 
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confidential ATSAP report was not captured in FAA's Air Traffic Quality 
Assurance (ATQA) database, and therefore, unknown to the air traffic facility 
management. Any losses of separation included in these unknown reports may not 
be in FAA's official count. At the same time, FAA's training for controllers and 
managers on the new policies and procedures has been limited. Facility officials 
said that while FAA training on new procedures was helpful for explaining the 
reasons for the changes to the procedures, it failed to explain the technical aspects 
of how errors should be reported. For example, facility officials said they received 
limited training on how to use a new database to store data on losses of separation. 
Managers at 15 of 25 of the air traffic facilities we visited 9 also raised concerns 
that the three ATO Service Areas may not have enough staff or knowledge of local 
flight procedures and airspace to effectively investigate operational errors. 

Recently, FAA developed corrective action plans to mitigate high-risk separation 
loss events — such as an aircraft executing an unexpected go-around 10 and aircraft 
arriving at the same altitude on parallel runways. However, because the Agency 
has not completed implementation of the action plans, it is too early to determine 
whether the plans will reduce the number of separation losses. In addition, FAA's 
corrective action plans do not include all safety risks identified by FAA and will 
not address all losses of separation that air traffic facility officials consider to be 
high risk. For example, FAA's process may not consider an instance of converging 
aircraft prevented from collision by Traffic Collision Avoidance Systems 
(TCAS) 11 to be a high-risk event if over 66 percent of the required separation was 
maintained. However, facility officials stated this type of event should be 
considered high risk regardless of the amount of separation loss because the 
controller introduced the risk of collision. 

We are making recommendations to improve FAA's policies and processes for 
identifying and mitigating separation losses. 

BACKGROUND 

At any given time, there are roughly 7,000 aircraft occupying U.S. airspace. To 
help maintain safe distances between aircraft, while under the control of air traffic 
controllers, FAA established minimum separation standards based on the aircraft's 
phase of flight and size. Controllers are responsible for providing instructions to 
pilots. 



Prior to January 30, 2012, ATQA contained reports of losses of separation. After this date, FAA began to replace 
ATQA with the Comprehensive Electronic Data Analysis and Reporting (CEDAR) database. 

9 These 25 facilities include: 9 ARTCCs, 1 1 TRACONs, 2 air traffic control towers, and 3 ATO Service Areas. 
Nationwide, there are a total of 21 ARTCCs (within the continental U.S), 28 TRACONS, 513 air traffic control towers, 
and 3 ATO Service Areas. Our selection criteria for facilities we visited are in Exhibit A. A listing of the facilities 
visited is in Exhibit B. 

10 A go around is an aborted landing of an aircraft on final approach to the runway. 

11 An onboard TCAS issues advisories for pilots to take evasive actions when the system detects a potential collision 
with another aircraft. 
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We have been reporting on separation losses for over a decade. Historically, 
FAA's oversight of operational error self-reporting has been problematic. Our 
previous work (see exhibit C) on operational errors has repeatedly raised concerns 
that nearly 300 FAA terminal facilities relied solely on controllers to self -report 
errors. In some cases, we found that the self-reporting process was subject to 
intentional manipulation. 12 More recently, in 2009, we found that inadequate 
guidance and insufficient staff contributed in part to control and oversight 
weaknesses in FAA's process for reporting and investigating losses. 

In response to our reports, FAA has undertaken a number of efforts to provide 
better oversight of and minimize separation losses, including the following: 

• FAA implemented the Risk Analysis Process (RAP), a new risk-based 
approach to address losses of separation. RAP consists of a panel of at least 
two controllers and a pilot. Three panels, one in each ATO Service Area, 
review separation losses for events in which pilots maintain less than 
66 percent of required separation. For example, if aircraft are required to be 
separated by 3 miles, a separation of less than 2 miles (66 percent of 3 miles) 
would be reviewed under RAP. FAA reviews various risk factors, including 
the severity and repeatability of the event, and identifies the highest risk 
events 13 to develop a list of the top five highest risk types of separation losses, 
along with corrective actions to address such hazards. 

• FAA revised its process for tracking high-risk errors. Previously, FAA 
tracked operational errors using a severity rating of A, B, or C — with A being 
the highest or most severe risk and C the lowest. The rating was based on the 
proximity of the aircraft to one another. 14 As a performance measure, prior to 
fiscal year 2011, FAA reported the rate of A and B errors per every thousand 
operations. However, in fiscal year 2011, FAA developed a new metric, the 
System Risk Event Rate (SRER), based on the high-risk events identified in 
RAP. This metric identifies the ratio between separation losses identified as 
high risk to all losses of separation. FAA' s goal is to limit the rate of high risk 
of standard separation losses to 20 or fewer for every 1,000 standard separation 
losses. 



For example, in both 2005 and again in 2008, our investigations at the Dallas/Fort Worth TRACON found that air 
traffic managers intentionally misclassified operational errors as either pilot deviations or "non-events" to reduce the 
number of operational errors reported at that location. Further, FAA's oversight processes failed to uncover this 
practice despite FAA's prior assurances that it would not allow operational errors to go unreported. 

13 Based on a risk assessment model used by the Eurocontrol, the European Organization for the Safety of Air 
Navigation. 

14 An "A" rating (high/severe risk) meant that less than 34 percent of separation standards were met; a "B" rating 
(moderate risk) at least 34 and less than 75 percent of separation standards were met; and a "C" rating (low risk) 

75 percent or more of separation standards were met, but the horizontal and vertical separation is less than 90 percent. 
Proximity events are minor losses of separation between two aircraft where 90 percent or greater of the required 
separation is maintained. 



5 



• FAA issued orders changing its process for identifying, investigating, and 
reporting losses of separation on January 30, 2012. One of the fundamental 
differences under the new orders is FAA's decision to move the responsibility 
for investigating losses of separation from specific air traffic facilities (where 
the losses in question occurred) to three ATO Service Areas. 15 

VARIOUS FACTORS CONTRIBUTED TO THE DRAMATIC 
INCREASE IN REPORTED OPERATIONAL ERRORS 

In the years leading up to fiscal year 2009, reported operational errors remained 
relatively stable (see figure 1). However, between fiscal years 2009 and 2010, 
reported operational errors increased 53 percent, from 1,234 to 1,887. FAA's 
recent numbers show a rise in the most serious Category A errors — from 37 in 
fiscal year 2009 to 43 in fiscal year 2010, and again to 55 in fiscal year 2011. 
Also, for fiscal years 2008, 2009, and 2010, FAA exceeded its target limits for the 
number of category A and B operational errors per million operations. 16 In 
addition, FAA did not meet its fiscal year 2011 goal of reducing high-risk events 
to no more than 20 per 1,000 separation losses; instead, the rate was 24.5 per 
1,000, or 23 percent higher than its goal. 

Figure 1. Operational Errors for Fiscal Years 2006 Through 201 1 
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Source: OIG analysis of FAA data. 



" FAA was prompted in part by our April 2008 recommendation to remove the quality assurance function at all Air 
Traffic Control facilities from the supervision of facility management. We made this recommendation because we 
reported in 2005 and 2008 that Dallas/Ft. Worth TRACON intentionally misclassified operational errors. See OIG 
Report No. CC-2007-083, "OIG Investigation-Alleged Cover-up of Operational Errors at DFW TRACON," April 18, 
2008. 

16 In fiscal year 2008, FAA goal was 2.15 A and B operational errors per million operations and the reported number 
was 2.25. In fiscal year 2009, the goal was 2.10 and the reported number was 2.44. In fiscal year 2010, the goal was 
2.05 and the reported number was 3.32. 
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A large percentage of the increases in operational errors reported between fiscal 
years 2009 and 2010 occurred at the same 10 facilities — 60 percent of the total 
increase of 653 errors (see table 1). TRACONs had the largest increase in reported 
errors — 86 percent. 



Table 1. Ten Air Traffic Facilities With the Largest Increase in 
Operational Errors From Fiscal Year 2009 to Fiscal Year 2010 



Facility 


FY 2009 


FY 2010 


Percent 
Increase 


Southern California TRACON 


33 


189 


473% 


Central Florida TRACON 


5 


24 


380% 


Houston TRACON 


11 


44 


300% 


Miami ARTCC 


15 


30 


1 00% 


Potomac TRACON 


21 


41 


95% 


Dallas/Fort Worth TRACON 


84 


143 


70% 


Charlotte Douglas International Airport Air Traffic Control Tower 


20 


34 


70% 


New York TRACON 


74 


119 


61% 


New York ARTCC 


25 


40 


60% 


Atlanta ARTCC 


35 


50 


43% 


Total 


323 


714 


121% 



Source: OIG analysis of FAA data. 



According to FAA, increased ATSAP reporting contributed to the 53 percent 
increase in reported operational errors between fiscal years 2009 and 2010. While 
non-punitive reporting may encourage more controllers to report errors, we 
identified other factors that contributed to the increase. Additionally, managers 
and controllers we spoke with at 7 of the 10 facilities attributed the increases to 
other factors. These factors include the following: 

• ARTCC automated reporting indicates an increase in actual errors. The 

number of errors reported at FAA's ARTCCs — which have had a full-time 
detection tool in use for many years — also increased from fiscal year 2009 to 
2010 (from 353 to 489, an increase of 39 percent). This suggests that a portion 
of the overall increase is due to more errors actually occurring, rather than 
being attributable entirely to improved reporting. 

• FAA reclassified 147 aircraft landings guided by the Southern California 
TRACON as operational errors. Originally, the landings were not classified 
as errors because the TRACON was operating under a waiver that allowed 
aircraft landing simultaneously to be closer than normally allowed. In 2010, 
the Air Traffic Safety Oversight Service revoked the waiver, citing safety 
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concerns, and subsequently reclassified aircraft landings that occurred under 
the waiver. This accounted for 23 percent of the increase. 

• Improved radar and voice replay tools have facilitated incident reviews, 
allowing them to more readily determine whether an operational error 
occurred. Fewer errors may have been discovered under prior incident reviews 
because of the difficulty obtaining data. 

• FAA has placed additional emphasis on ensuring correct phrasing when 
pilots read back controllers' instructions. For example, FAA officials stated 
that greater emphasis has been placed on ensuring correct readbacks for visual 
separation procedures. If the controller fails to catch and correct inadequate 
readbacks, the event is reported as an operational error. Previously, these types 
of events were not reported as rigorously. 

EFFECTIVENESS OF FAA'S POLICIES AND PROCEDURES TO 
COLLECT, INVESTIGATE, AND REPORT SEPARATION LOSSES 
IS LIMITED 

FAA's policies and procedures to identify and report on losses of separation are 
limited by incomplete data and implementation challenges. For example, FAA 
lacks an accurate baseline of the actual total number of separation losses that 
occur. In addition, facility managers expressed concerns that FAA's training is not 
comprehensive in explaining how to use the new operational error reports database 
under the new procedures. Facility staff are also concerned about FAA's decision 
to move the responsibility for investigating errors to ATO Service Areas, which 
may not have adequate staffing levels and familiarity of each facility's operations. 

Data Collection Deficiencies and Reporting Inconsistencies Limit 
FAA's Efforts To Track Separation Losses 

FAA does not have an accurate baseline of separation losses due to gaps in TARP 
and ATSAP reporting and inconsistent classifications of losses. A complete 
picture of losses that occur is critical for FAA to track, fully address, and mitigate 
the risk of separation losses in the National Airspace System. 

In 2009, we recommended that FAA fully implement TARP at its air traffic 

17 

facilities. Although TARP has been installed and running at air traffic terminal 
facilities, 18 FAA does not analyze and report all separation losses that are 
automatically flagged by TARP. TARP provides alerts when a potential loss of 
separation occurs. Then, FAA must investigate the TARP alerts to determine 



OIG Report No. AV-2009-045, "FAA's Process for Reporting and Investigating Operational Errors", March 24, 
2009. 

18 Terminal facilities are TRACONs and airport control towers. 
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whether an actual loss of separation happened. FAA Service Area staff only 
investigate losses of separation of less than 70 percent. 19 

In addition, ATSAP data indicate that more losses of separation might be 
occurring than those that are known to facility management. As we reported in 

20 21 

2012, approximately 50 percent of all ATSAP event reports are classified as 
"unknown," which means that the event captured in the confidential ATSAP 
report was not captured in FAA's Air Traffic Quality Assurance (ATQA) 
database, 22 and therefore, unknown to the air traffic facility management. Any 
losses of separation included in these unknown reports may not be in FAA's 
official count. 

Inconsistencies in FAA's classification of separation losses further reduce the 
reliability of the Agency's count of operational errors and runway incursions. For 
example, in fiscal year 2011, there were a total of 1,895 reported operational 
errors, a negligible increase from the 1,887 reported in fiscal year 2010. However, 
the fiscal year 2011 count does not include 157 events classified as runway 
incursions that occurred in August 2011 at Charlotte-Douglas International 
Airport. FAA reclassified the incursions as non-events due to its interpretation of 
the definition of a runway incursion and the judgment by senior Agency officials 
that safety was not compromised. 24 Runway incursions are considered to be 
operational errors when an air traffic controller is the cause. Therefore, had the 
classification to non-events not occurred, the fiscal year 2011 operational error 
count would have increased by 9 percent of the number of reported errors the 
previous year. 

Staffing Challenges Undermine FAA's Efforts To Investigate 
Separation Losses 

FAA faces significant staffing challenges with implementing its new procedures 
for investigating separation losses. Prior to January 30, 2012 — when FAA 
implemented the new procedures — investigations were conducted at the Nation's 
more than 300 air traffic facilities. Therefore, conservatively, at least 300 staff 
conducted investigations on at least a part-time basis. With the implementation of 



Losses of separation in which less than 70 percent of the separation was maintained. 

20 OIG Report Number 2012-152, "Long-Term Success of ATSAP Will Require Improvements in Oversight, 
Accountability, and Transparency," July 19, 2012. 

21 Event reports include identify actual or potential losses of separation, including operational errors, or other situations 
that may degrade air traffic safety. 

22 Prior to January 30, 2012, ATQA contained reports of losses of separation. After this date, FAA began to replace 
ATQA with the Comprehensive Electronic Data Analysis and Reporting (CEDAR) database. 

23 The definition of a runway incursion is any occurrence at an airport involving "the incorrect presence of an aircraft, 
vehicle, or person on the protected area of a surface designated for the landing or take-off of aircraft." 

24 The errors involved the continued clearance of 157 take-offs and landings on a runway that was in close proximity to 
a disabled commercial airplane that had previously aborted a takeoff and was cleared off the runway onto an adjacent 
taxiway for maintenance. A portion of the disabled airplane intruded upon the protected area of the active runway 
environment. 
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FAA's new procedures, the number of personnel investigating losses of separation 
has been substantially reduced. Currently, FA A has a total of 16 staff in the three 
Service Areas to investigate the high numbers of separation losses that occur. 
However, the Agency plans to hire additional staff. 

FAA faces substantial staffing challenges in several areas, including: 

• Increased workload: Once TARP data from all facilities with TARP are 
reviewed by Service Areas, FAA estimates that an additional 600 to 900 
reports of separation losses will need to be reviewed each day. 

• Available tools: FAA has not fully implemented the Digital Audio Legal 
Recorder, which allow staff in the Service Areas to obtain air traffic audio data 
remotely and review audio replays of incidents. Without this automated tool, 
staff must request the facility to send the audio data for each incident to their 
Service Area, which could delay investigations of separation losses. FAA is 
aware of this issue and plans to address it as implementation continues. 

• Training: Some facility officials stated that the FAA training on new 
procedures was helpful for explaining the reasons for the changes to the 
procedures, but failed to explain the technical aspects of how things should be 
done. For example, as part of its new procedures, FAA replaced its previous 
ATQA reporting database with a new database called Comprehensive 
Electronic Data Analysis and Reporting (CEDAR) for completing loss of 
separation reports. Yet facilities received limited training on how to use the 
new database. Additionally, FAA acknowledged that a lack of facility training 
on how to properly complete operational error reports contributed to 
incomplete or inadequate causal data in its ATQA database. In ATQA, 
descriptions in the causal section of these reports often provided limited 
information about the cause of the errors. For example, according to FAA, the 
causal factor called "inappropriate use of displayed data [on aircraft]" was 
identified as the cause in 50 percent of all ATQA reports. This description of 
cause did not always identify the cause of the incident such as a training 
deficiency, fatigue, or a lack of experience. 

• Knowledge of local facility operations: Air Traffic facility officials 
questioned whether the Service Center staff have knowledge of each facility's 
local procedures and operations such as the expertise and technical knowledge 
related to aircraft routing and altitudes covered by a facility, particularly in 
complex areas in large cities. 
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FAA'S NEW CORRECTIVE ACTION PLANS DO NOT ADDRESS ALL 
SAFETY CONCERNS RELATED TO SEPARATION LOSSES 

FAA has developed a new strategy to mitigate separation losses — including 
implementing corrective action plans to reduce the five highest risk events. 
However, FAA does not examine all losses of separation that air traffic officials 
considered to be high risk. Facility officials also question the effectiveness of 
FAA's mitigation efforts. 

FAA's Corrective Action Plans Do Not Include Safety Risks and 
Causal Factors Identified in Previous FAA Analyses 

In fiscal year 2011, FAA developed new corrective action plans to reduce the five 
highest risk separation loss events identified through RAP. For example, two of 
the top five risk categories involve loss of separation when an aircraft overshoots 
the turn to final approach on parallel runways and conflicts with another aircraft. 
FAA's plans to mitigate these risks include the following: 

• limit the turn to final angle of no more than 30 degrees (see figure 2); 

• require altitude separation of at least 1,000 feet prior to application of visual 
separation; and 

• develop airport- specific speed restrictions for final approach courses. 



11 



Figure 2. Example of a Corrective Action Plan Initiative 
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FAA also identified factors contributing to the top five high-risk events (see 
table 2). 
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Table 2. Overview of Top Five Risk Categories and Corrective 
Action Plans 



Top Five Event 
Category 


Causal Factors 


Key Corrective Action Plan 
Initiatives 


1 . Arrival aircraft 
executes an 
unexpected go around 


Lack of adequate go around 
procedures 


Each facility must define local 
procedures for go arounds. 


2. Arriving aircraft at 
the same altitude on 
parallel runways. 


Aircraft overshoot the turn to final 
and conflict with parallel traffic 
same altitude. 


Proposal to require aircraft on flight 
paths at right angles to the landing 
runway to be assigned altitudes at 
least 1 ,000 feet apart until visual 
separation is established. 


3. Aircraft at an 
altitude other than 
expected. 


(a) Breakdown of communications 

(b) Incorrect data block altitudes 
prevent conflict alert activation 


Investigate the feasibility of using 
"Mode S" technology to allow 
controllers to view the altitudes 
pilots have entered into aircraft 
svstpms 


4. Aircraft in 
unexpected position 
resulting in a loss of 
separation. 


Controller Coordination: Aircraft 
transferred to another controller on 
route or at different altitude than 
expected. 


Develop training to address 
coordination. 


5. Aircraft vectored at 
speed and/or angle of 
intercept, leading to 
loss of separation. 


Current air traffic procedures 
(71 1 0.65) allow aircraft to turn to 
final approach at excessive speed 
and excessive intercept angle. 


Proposed procedure changes to: 
(1) require headings that allow an 
intercept angle to final approach of 
no greater than 30 degrees, and (2) 



require development of local speed 
restrictions for facilities that vector to 
final approach on parallel runways. 



Source: FAA. 

However, FAA's corrective action plans, which were finalized in July 2011, do 
not address previously identified causal factors, trends based on the factors, or 
follow-up actions to address them. In particular, in June 2011, RAP identified 
perception, memory, training, and experience as the top causal factors for high- 
risk events. Also, the RAP panel identified training and on-the job training as key 
corrective action areas and recommended FAA: 

• Develop specific training requirements and curriculum to address poor 
recovery. Of all events analyzed through RAP, 41 percent involved poor 
recovery from a loss of separation. The RAP panel also found over half of the 
recovery efforts worsened the situation, causing the aircraft to be in closer 
proximity. Furthermore, RAP determined that of the 87 most severe risk 
analysis events, 76 (or 87 percent) involved poor air traffic control recovery. 
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• Conduct root cause analyses and determine the best way to address 
incidents that occur during on-the-job training. These incidents accounted 
for 10 percent of all analyzed events. 

FAA chose to exclude these causal factors (perception, memory, training, and 
experience) and trends (poor recovery and on-the-job training) from the corrective 
action plans because it considered them to be "contextual." However, a 2005 study 
published by FAA found that the majority (86 percent) of operational errors 
reviewed were "skill-based errors," which tend to be "the result of habitual actions 
associated with an individual's attention, memory, and/or execution technique." 
The study's finding is similar to causal factors identified by FAA in its 2011 RAP 
document. Also, the trend of poor recovery was present in the majority of the most 
severe risk analysis events reviewed. 

Facility Officials Question the Effectiveness of Mitigation Efforts 

Facility officials questioned the overall effectiveness of FAA's corrective action 
plans. While the new air traffic procedures proposed in the plans are reviewed for 
risk through the Safety Management System 26 (SMS) process, facility officials had 
concerns with the proposed procedures, as shown in table 3 below. 

Table 3. FAA Proposals To Mitigate Losses of Separation and Facility 
Officials' Concerns With Them 



FAA Proposals for Facilities 

Determine facility-specific speed requirements 
to reduce the potential of an aircraft 
overshooting a runway 

Determine the feasibility of the potential benefits 
from automatically relaying additional altitude 
information from the aircraft's autopilot to 
controllers 

Define procedures for go-arounds and missed 
approaches 



Facility Officials' Concerns 

Controllers should let aircraft fly the most 
efficient approach, unless speed control for 
separation is required. 

FAA must consider the impact of this 
information on controller workload. 



Controllers must assess the situation and make 
the best decision based on established 
procedures and other traffic. Facilities cannot 
plan for every scenario, because varying traffic 
flows can affect the best option for controllers. 



Source: FAA and OIG analysis. 

Finally, air traffic facility officials are concerned that FAA's RAP, SRER, and 
corrective action plans do not include events they consider to be high-risk, such as 
those for which the loss of separation was more than 66 percent. For example, 
converging aircraft prevented from colliding by the Traffic Collision and 



"Examining ATC Operational Errors Using the Human Factors Analysis and Classification System," 
DOT/FAA/AM-05/25, FAA Office of Aerospace Medicine, December 2005. 

26 FAA defines the Safety Management System as an integrated collection of processes, procedures, policies, and 
programs that are used to assess, define, and manage the safety risk in providing ATC and navigation services. 
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Avoidance System (TCAS) could be a high-risk event regardless of the degree of 
separation loss. TCAS, an automated cockpit avoidance system, operates 
independently from the ground-based Air Traffic Control system and serves as a 
last line of defense for preventing mid-air collisions. In other words, use of the 
TCAS system could indicate a high-risk event because it is an event in which the 
controller failed to separate aircraft and introduced the risk of collision. Yet 
FAA's mitigation plans do not include all of these events. 

CONCLUSION 

Reported losses of separation continue to be a major air safety concern, 
particularly in light of dramatic increases in their occurrence. While FAA recently 
issued new policies and processes for investigating and mitigating separation 
losses, a lack of a reliable baseline creates substantial challenges for FAA to 
ensure these new policies and processes are effective. Until FAA takes action to 
determine the true magnitude of operational errors, assess their potential safety 
impacts, identify their root causes, and align adequate staffing for oversight, the 
risk of separation losses will remain a safety concern. 

RECOMMENDATIONS 

To improve its policies and processes for collecting, investigating, and reporting 
separation losses, and mitigating their risks, we recommend that FAA: 

1. Include all losses of separation that are reported under ATSAP, but unknown 
to air traffic facilities, in its official count of such events. 

2. Determine the level of staff and expertise needed at the ATO Service Areas to 
effectively implement ATO's new Orders on investigating losses of separation, 
audit all TARP data, and initiate actions to fill those requirements. 

3. Determine the extent to which ATO has successfully implemented its new 
orders (effective January 2012). This determination should include reviews of 
the quality of separation loss investigation reports, effectiveness of training, 
and additional actions or resources needed. 

4. Include high-risk TCAS warning events in its Risk Analysis Process and 
System Risk Event Rate when the separation between two converging aircraft 
is maintained at 66 percent or more. 

5. Develop actions to mitigate the following situations identified in the Risk 
Analysis Process: (1) poor recovery from loss of separation and (2) losses of 
separation involving on-the-job training. 
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6. Utilize analysis of the causal and contributory factors derived in the Risk 
Analysis Process — including perception, memory, and training — to identify the 
underlying reasons for separation losses and develop mitigation strategies to 
address those causes. 

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 
RESPONSE 

We provided FA A with a draft copy of this report on December 14, 2012, and 
received FAA's response on January 25, 2013. FAA's response is included in its 
entirety in the appendix to this report. In its response, FAA fully concurred with 
recommendations 2, 3, 5, and 6, and partially concurred with recommendations 1 
and 4. FAA's planned actions for recommendations 2, 3, 4, and 5 met the intent of 
our recommendations and included reasonable timeframes for implementation. 
However, we are requesting that FAA reconsider its response or provide additional 
information for recommendations 1 and 6, as detailed below. 

For recommendation 1, FAA stated that there would be legal and data 
compatibility concerns with integrating ATSAP-reported losses of separation into 
its baseline of total separation losses. Further, the Agency stated that TARP 
captures the vast majority of separation losses and has proven to be the best 
solution for establishing a new baseline. We agree that the process of fully 
implementing TARP has generated significantly more data than FAA previously 
had available. However, FAA is assuming that there is a virtually 100 percent 
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overlap between ATSAP-reported losses and TARP-detected losses, which we 
question as we are not aware of any effort by FAA to reconcile ATSAP and TARP 
data. Without this reconciliation, FAA cannot be assured that ATSAP data are 
being captured in the Agency's counts of losses of separation. FAA also did not 
provide a rationale for its exclusion of these data, nor has it clarified the legal 
impediments to integrating ATSAP-reported losses of separation into its total 
count of separation losses. Furthermore, there are several air traffic facilities with 
equipment that is not compatible with TARP, including in Honolulu, HI; San Juan, 
PR; Twin Falls, ID; and Helena, MT. To determine losses of separation at these 
locations, FAA may rely on manual reporting by controllers, which is less 
effective, as we have pointed out in our prior work. Therefore, in our opinion, 
FAA is missing an opportunity to establish a complete and accurate baseline of 
separation losses by not including ATSAP-reported losses in its count, and we 
request it reconsider its position on this recommendation. 



FAA does not analyze and report all separation losses that are flagged by TARP. FAA Service Area staff investigate 
losses of separation identified by TARP of less than 70 percent (i.e., when less than 70 percent of the separation 
between aircraft was maintained). 
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For recommendation 6, FAA stated that it believes it has met the intent of our 
recommendation because its Risk Analysis Process (RAP) and yearly 
identification of the Top 5 hazards in the NAS use both causal and contributory 
factors to develop corrective actions and mitigation strategies. However, FAA did 
not specify how its processes have used the specific causal and contributory 
factors we included in our recommendation — perception, memory, and training. 
As we noted in our report, FAA chose to exclude these factors from its corrective 
action plans because it considered them to be "contextual." Therefore, to ensure 
that FAA's response meets the full intent of our recommendation, we request that 
FAA provide us with more information on how RAP uses these factors in its 
mitigation strategies. 

ACTIONS REQUIRED 

We consider recommendations 2, 3, 4, and 5 resolved but open pending the 
completion of the planned actions. We also consider recommendations 6 resolved 
but open pending receipt of supporting documentation of FAA's actions taken. For 
recommendation 1, we request that FAA reconsider its position. In accordance 
with Department of Transportation Order 8000. 1C, we request that FAA provide 
us this additional information within 30 days. 

We appreciate the courtesies and cooperation of FAA representatives during this 
audit. If you have any questions concerning this report, please call me at 
(202) 366-0500 or Scott Macey, Program Director, at (415) 744-0434. 

# 

cc: DOT Audit Liaison, M-l 

FAA Audit Liaison, AAE-100 
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EXHIBIT A. SCOPE AND METHODOLOGY 

We conducted this audit in accordance with generally accepted Government 
auditing standards. Those standards require that we plan and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that the 
evidence obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. We conducted this performance audit from June 
2011 through December 2012, which included site visits to FAA Headquarters and 
air traffic facilities nationwide (see exhibit B). 

To evaluate the effectiveness of FAA's policies and procedures to collect, 
investigate, and report separation losses, we interviewed FAA Headquarters 
officials and analyzed FAA orders, data, and reports on losses of separation. In 
addition, we interviewed officials and analyzed information at the 30 facilities we 
contacted to evaluate these issues. We visited or contacted 9 of 21 nationwide Air 
Route Traffic Control Centers (ARTCC) within the continental United States, 
11 of 28 stand-alone Terminal Radar Approach Control (TRACON) facilities, 2 of 
513 Air Traffic Control Towers, 4 of 81 Flight Standards District Offices 
(FSDO), 28 3 of 3 ATO Service Areas, and 1 of 20 Certificate Management Offices 
(CMO). Specifically, the OIG statistician selected a stratified random sample of 
11 (6 TRACON and 5 ARTCC) out of 41 TRACON and ARTCC facilities that 
had increases in operational errors from fiscal years 2009 to 2010. We focused on 
ARTCCs and TRACONs in our sample because these facilities had the largest 
increases in operational errors between fiscal years 2009 and 2010. Over this time 
period, errors at TRACONs increased 86 percent while errors at ARTCCs 
increased 39 percent. We visited 10 of these facilities and had a teleconference 
with the remaining facility. 30 We also visited 19 other facilities to obtain a broader 
range of information about operations nationwide including facilities located near 
our sample sites. A list of the sample sites and additional sites we visited in is 
exhibit B. 

To determine various factors that contribute to increases in reported losses of 
separation, we interviewed FAA Headquarters officials as to why operational 
errors increased from fiscal year 2009 to 2010. We also interviewed officials and 
analyzed information at the 30 facilities we contacted to identify any other 
contributing factors. 

To assess the effectiveness of FAA's efforts to mitigate operational errors, we 
analyzed mitigations and corrective action plans from FAA Headquarters. 



FSDOs are part of FAA's Flight Standards Service, which promotes safe air transportation by setting the standards 
for certification and oversight of airmen, air operators, and air agencies. 

29 CMOs specialize in areas such as the certification, surveillance, and inspection of major air carriers. 

30 We teleconferenced with Anchorage Air Route Traffic Control Center. 



Exhibit A. Scope and Methodology 
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Furthermore, we obtained and reviewed mitigation strategies at the 30 facilities we 
contacted. At both FAA headquarters and the 30 contacted facilities, we 
interviewed officials for their perspective on operational error mitigation 
strategies. 



Exhibit A. Scope and Methodology 
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EXHIBIT B. ORGANIZATIONS VISITED OR CONTACTED 

Washington Air Route Traffic Control Center (ZDC) 
Seattle Air Route Traffic Control Center (ZSE) 
Fort Worth Air Route Traffic Control Center (ZFW) 
Atlanta Air Route Traffic Control Center (ZTL)* 
New York Air Route Traffic Control Center (ZNY)* 
Chicago Air Route Traffic Control Center (ZAU) 
Cleveland Air Route Traffic Control Center (ZOB)* 
Boston Air Route Traffic Control Center (ZBW)* 
Anchorage Air Route Traffic Control Center (ZAN)* 

Potomac Consolidated Terminal Radar Approach Control (PCT) 
Seattle Terminal Radar Approach Control (S46) * 
Northern California Terminal Radar Approach Control (NCT) 
Southern California Terminal Radar Approach Control (SCT)* 
Houston Terminal Radar Approach Control (190)* 
Dallas/Fort Worth Terminal Radar Approach Control (DIO)* 
Las Vegas Terminal Radar Approach Control (L30)* 
New York Terminal Radar Approach Control (N90)* 
Chicago Terminal Radar Approach Control (C90) 
Cleveland Terminal Radar Approach Control (CLE) 
Boston Consolidated Terminal Radar Approach (A90) 

Seattle-Tacoma Air Traffic Control Tower (SEA) 

Los Angeles International Air Traffic Control Tower (LAX) 

Dallas Fort Worth Flight Standards District Office (DFW FSDO) 
Atlanta Flight Standards District Office (ATL FSDO) 
New York City Flight Standards District Office (NYC FSDO) 
Chicago OHare Flight Standards District Office (ORD FSDO) 

Delta Certificate Management Office (Delta CMO) 

FAA Western Service Area 
FAA Central Service Area 
FAA Eastern Service Area 

*This facility was part of our 1 1 facility sample. 



Exhibit B. Organizations Visited or Contacted 
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EXHIBIT C. DOT OIG REPORTS ON OPERATIONAL ERRORS 
SINCE 2000 



Issue Date and Title Key Findings 



March 24, 2009 
FAA 's Process for 
Reporting and 
Investigating Operational 
Errors (AV-2009-045) 



April 18, 2008 
OIG Investigation- 
Alleged Cover-up of 
Operational Errors at 
DFW TRACON 
(CC-2007-083), 



Control and oversight weaknesses in FAA's process for reporting and 
investigating losses of separation caused by pilots and controllers. 
These weaknesses were due in part to inadequate FAA guidance for 
investigating these events and insufficient staffing in the Air Traffic 
Organization (ATO) Safety Office. Further, FAA's processes did not 
ensure that all losses of separation were accurately reported across 
terminal and en route facilities or consistently evaluated for severity. 

Air traffic managers at the TRACON intentionally misclassified 
operational errors as either pilot deviations or "non-events" to reduce 
the number of operational errors reported at that location. FAA's 
oversight processes failed to uncover this practice despite FAA's prior 
assurances that it would not allow operational errors to go 
unreported. 



February 14, 2005 
Alleged Cover-up of 
Operational Errors at 
DFW TRACON 
(CC-2004-067) 



FAA DFW TRACON managers had failed to investigate, and 
therefore, underreported operational errors over a seven-year period. 
In short, management created an atmosphere of self-reporting and 
other incentives that discouraged employees from identifying 
operational errors. 



September 20, 2004 
Controls Over the 
Reporting of Operational 
Errors (AV-2004-085) 



At facilities that handle the most air traffic, FAA relies on supervisors 
and controllers to self-report when errors have occurred and does not 
have a system in place to verify that this reporting process is reliable. 



April 3, 2003 
Operational Errors and 
Runway Incursions 
(AV-2003-040) 

December 15, 2000 
Actions To Reduce 
Operational Errors and 
Deviations Have Not 
Been Effective 
(AV-2001-11) 



Despite FAA progress in reducing operational errors and runway 
incursions, the number of incidents remained high. On average, one 
runway incursion and three operational errors occurred each day in 
fiscal year 2002. High-risk incursions and errors occurred, on 
average, once every 10 days and 8 days, respectively. 

FAA does not determine the severity of operational errors, and 
controllers and managers have different perceptions of actions taken 
when operational errors occur. According to the National Air Traffic 
Controllers Association, controllers face serious disciplinary actions 
for committing operational errors. Facilities with the most reported 
operational errors over the previous 5 fiscal years lacked adequate 
plans to reduce operational errors. Further FAA has not provided 
provide strong national oversight to effectively reverse the upward 
trend in operational errors. 



Exhibit C. DOT OIG Reports on Operational Errors Since 2000 
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EXHIBIT D. MAJOR CONTRIBUTORS TO THIS REPORT 

Name Title 



Scott Macey 
Stephen Jones 
Kevin Montgomery 
Judy Nadel 
Susan Crook 
Karen Sloan 
Andrea Nossaman 
Audre Azuolas 
Petra Swartzlander 
Megha Joshipura 



Program Director 
Project Manager 
Senior Analyst 
Senior Auditor 
Analyst 

Communications Officer 
Senior Writer 
Writer-Editor 
Senior Statistician 
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Exhibit D. Major Contributors to This Report 
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ffii &m Federal Aviation 
V^fflBj Administration 



Memorandum 



Date: 



January 25, 2013 



From: 



To: 




Subject 



Response to Office of the Inspector General (OIG) Review of FAA's Efforts to 
Track and Mitigate Air Traffic Losses of Separation 



The Federal Aviation Administration (FAA) through its Air Traffic Organization (ATO) is 
committed to conducting safe operations throughout the Nation Airspace System (NAS). 
Tracking data relating to aircraft operations that do not maintain standard separation distances is 
an important component of risk assessment in the NAS and is used to develop appropriate 
corrective actions when significant risks are identified. 

In January of 2012, the ATO implemented a significant change in the way safety data, including 
losses of separation, are reported, analyzed, and acted upon. These changes improve utilization 
of the Air Traffic Safety Action Program (ATSAP), which is a non-punitive reporting system, as 
well as the Traffic Analysis and Review Program (TARP), which electronically identifies losses 
of separation. As a result, the ATO has seen a dramatic increase in reporting and now has 
available unprecedented amounts of both qualitative safety data, through ATSAP, and 
quantitative data from TARP. The validation and analysis processes that have been implemented 
have greatly enhanced the agency's ability to identify and prioritize high risk hazards, which are 
addressed through the ATO's Top 5 program. This program provides a forum for a coordinated 
and collaborative approach to identify and mitigate risks through the most effective means 
available, including training, procedures, and technological improvements. 

The ATO continues to implement enhancements in training, procedures, and technology based 
on its use of a dramatically improved reporting, with many new initiatives planned throughout 
2013, which offer the potential to further enhance the safety of operations in the NAS. 

RECOMMENDATIONS AND RESPONSES 

OIG Recommendation 1 : Include all losses of separation that are reported under ATSAP, but 
unknown to air traffic facilities, in its official count of such events. 

FAA Response : Partially-concur. The FAA agrees that knowledge of operational incidents is 
crucial to its ability to identify and address safety trends. In order to achieve that goal FAA 
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successfully developed and is in the process of fully implementing the TARP system that 
captures quantitative data relating to the vast majority of occurrences that involve loss of 
separation. TARP was fully implemented at all terminal radar facilities in September 2012. 
FAA is now developing TARP for En Route facilities, and with implementation targeted for May 
2013. TARP has already demonstrated its capability to operate effectively in the terminal 
environment, and is generating ten times the amount of data FAA previously had available 
relating to loss of separation. Further, the system increases the utility of the data captured by 
consolidating the information into a single database available to all facilities. Increased reporting 
is a key strategy to improve air traffic safety and TARP has already proven to be the best 
solution for the FAA to establish a new baseline of reporting losses of separation. Incorporating 
ATSAP data, as recommended would bring about significant legal issues with regard to the use 
of this information and introduce considerable data compatibility problems. As a result, FAA is 
focused on fully implementing and utilizing the vast amount of new data made available by 
TARP, which provides the most accurate available metric for these occurrences. 

OIG Recommendation 2 : Determine the level of staff and expertise needed at the ATO Service 
Areas to effectively implement ATO's new Orders on investigating losses of separation, audit all 
TARP data, and initiate actions to fill those requirements. 

FAA Response : Concur. The FAA Air Traffic Organization (ATO) has just completed the 
largest and most significant improvements in the last 30 years to the way air traffic control risk 
and safety performance are managed in the United States. From the implementation of voluntary 
reporting, to electronic detection, the development of standardized risk assessment processes and 
the establishment of a proactive safety management system, the ATO is now able to identify 
precursors to risk rather than just react to single incidents. Our processes and metrics need to 
mature before a proper baseline can be established. We have made commitments to the Office of 
the Secretary and the Office of Management and Budget that at the end of FY14 we will have 
established sound baseline for metrics based upon about two years' worth of data. With the 
experience developed over that period, FAA will be better able to understand the staffing 
requirements for making full and appropriate use of this newly available data. While FAA will 
continue to evaluate informally resource requirements over the implementation period, it intends 
to conduct a formal staffing study for TARP management and analysis by October 1, 2015. 

OIG Recommendation 3 : Determine the extent to which ATO has successfully implemented 
its new orders (effective January 2012). This determination should include reviews of the quality 
of separation loss investigation reports, effectiveness of training, and additional actions or 
resources needed. 

FAA Response : Concur. The ATO combined its Safety and Technical Training offices into one 
service unit in 2012 under the leadership of the Vice President, Safety and Technical Training. 
This combination was initiated for the specific purpose of identifying safety issues, evaluating 
the effectiveness of training, and collaboratively identifying necessary resources (e.g. training, 
staffing, procedures, technological improvements) to improve overall safety in the delivery of air 
traffic control services. 
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The Safety directorate's QA Group is in the final stage of developing a Standard Operating 
Procedure (SOP) to address the validation and processing of MORs and EORs for each of the 
QA Service Area offices. The QA SOP will be completed in early 2013 and subsequent training 
of these procedures will be conducted throughout 2013. 

The Safety directorate has initiated the development of a revised QA and Quality Control (QC) 
Training Course for ATO personnel to be taught at the FAA Academy. Coordination is currently 
ongoing between the Safety and Technical Training directorates to fully define requirements. 
This course will include lessons on how to utilize Comprehensive Electronic Data Analysis 
Reporting (CEDAR) and provide an overall understanding of the QA and QC processes that have 
been implemented. 

Additionally, QA staff has conducted visits to the QC Groups in each of the ATO Service Areas 
to provide clarification and familiarization with the new Orders, CEDAR and TARP. In support 
of these visits, the ATO is in the process of publishing Safety Guidance reflecting best practices 
on how to implement the new orders. This guidance is expected to be published by February 
2013. In addition, the Air Traffic Oversight Service is planning to audit the implementation of 
the new Orders in fiscal year (FY) 2013. 

FAA will use the results of these incremental compliance and effectiveness evaluations of the 
individual components of its new approach to continue fine tuning its operation of individual 
elements of its new safety scheme. It intends to conduct an overall evaluation of the overall 
scheme by October 1, 2015, once sufficient data is available to meaningfully evaluate its 
performance. 

OIG Recommendation 4 : Include high-risk TCAS warning events in its Risk Analysis Process 
and System Risk Event Rate when the separation between two converging aircraft is maintained 
at 66 percent or more. 

FAA Response : Partially-concur. The ATO has a well-developed risk analysis process that 
reviews events on a prioritized basis in accordance with the risk factors present during an event. 
TCAS is one of several safety countermeasures, and its effectiveness or lack thereof is already 
part of the analysis process used by the ATO. The ATO will clarify and produce written criteria 
for determining which TCAS events represent a high-risk and should be included in the risk 
assessment process. It intends to complete these criteria by December 30, 2013. 

OIG Recommendation 5 : Develop actions to mitigate the following situations identified in the 
Risk Analysis Process: (1) poor recovery from loss of separation and (2) losses of separation 
involving on-the-job training. 

FAA Response : Concur. The ATO is already addressing these issues. The number one hazard 
identified in the FY 2013 Top 5 hazards in the NAS was poor recovery after a loss of separation. 
As a result, FAA- wide corrective action and monitoring plans have been developed. Hazard 
Risk Mitigation is a high priority DOT goal that tracks how well the FAA is doing in 
implementing corrective actions on a yearly basis. To meet this goal the FAA has to complete a 
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minimum of 80% of approved mitigations to address the top five contributing hazards to high 
risk events every year. 

The ATO does not have data that directly ties the on-the-job training (OJT) program to a specific 
increase in risk for losses of separation. However, the ATO plans to gather more information 
about such incidents in order to determine whether there is sufficient support provided to OJT 
instructors in the current training methodology. The ATO will be amending CEDAR to require 
identification of OJT in progress when operational incidents occur. ATO intends to implement 
this upgrade by June 2013. 

OIG Recommendation 6 : Utilize analysis of the causal and contributory factors derived in the 
Risk Analysis Process — including perception, memory, and training-to identify the underlying 
reasons for separation losses and develop mitigation strategies to address those causes. 

FA A Response : Concur. RAP and the yearly identification of the Top 5 hazards in the NAS 
utilize both causal and contributory factors to develop corrective actions and mitigation strategies 
to address causes for losses of separation in the NAS. Following our Safety Management 
System which requires continuous improvement of our processes, the ATO is making 
improvements to RAP and other safety programs, such as Search and Rescue and QC, as well as 
sharing the ATO's safety data with Aviation Safety Information Analysis and Sharing for 
analysis of air traffic control and aircraft data. Combining air traffic and aircraft data offers 
numerous opportunities to improve aviation safety. The FAA believes it has met the intent of 
this recommendation and requests that it be closed. 
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